Entropy of a random variable

Consider RV $F={f_1,f_2,...,f_K}$, with probability $p_k=\mathrm{Prob.}\{F= f_K\}$

Self-Information of one realization $f_k : H_k= -\log(p_k)$

• $p_k=1$: always happen, no information
• $P_k \sim 0$: seldom happen, its realization carries a lot of information

Entropy = average information $$H(\mathcal{F})=-\sum_{f \in \mathcal{A}} p_{\mathcal{F}}(f) \log_2 p_{\mathcal{F}}(f)$$

• Entropy is a measure of uncertainty or information content, unit=bits
• Very uncertain -> high information content